High expression of glutaminyl cyclase (QC) contributes to the initiation of Alzheimer's disease (AD) by catalyzing the generation of neurotoxic pyroglutamate (pE)-modified β-amyloid (Aβ) peptides. Preventing the generation of pE-Aβs by QC inhibition has been suggested as a novel approach to a disease-modifying therapy for AD. In this work, a series of diphenyl conjugated imidazole derivatives (DPCIs) was rationally designed and synthesized. Analogues with this scaffold exhibited potent inhibitory activity against human QC (hQC) and good in vitro blood-brain barrier (BBB) permeability. Further assessments corroborated that the selected hQC inhibitor 28 inhibits the activity of hQC, dramatically reduces the generation of pE-Aβs in cultured cells and in vivo, and improves the behavior of AD mice.